{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<i>Copyright (c) Microsoft Corporation. All rights reserved.</i>\n",
    "\n",
    "<i>Licensed under the MIT License.</i>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# DKN : Deep Knowledge-Aware Network for News Recommendation\n",
    "\n",
    "DKN \\[1\\] is a deep learning model which incorporates information from knowledge graph for better news recommendation. Specifically, DKN uses TransX \\[2\\] method for knowledge graph representation learning, then applies a CNN framework, named KCNN, to combine entity embedding with word embedding and generate a final embedding vector for a news article. CTR prediction is made via an attention-based neural scorer. \n",
    "\n",
    "## Properties of DKN:\n",
    "\n",
    "- DKN is a content-based deep model for CTR prediction rather than traditional ID-based collaborative filtering. \n",
    "- It makes use of knowledge entities and common sense in news content via joint learning from semantic-level and knowledge-level representations of news articles.\n",
    "- DKN uses an attention module to dynamically calculate a user's aggregated historical representaition.\n",
    "\n",
    "\n",
    "## Data format\n",
    "\n",
    "DKN takes several files as input as follows:\n",
    "\n",
    "- **training / validation / test files**: each line in these files represents one instance. Impressionid is used to evaluate performance within an impression session, so it is only used when evaluating, you can set it to 0 for training data. The format is : <br> \n",
    "`[label] [userid] [CandidateNews]%[impressionid] `<br> \n",
    "e.g., `1 train_U1 N1%0` <br> \n",
    "\n",
    "- **user history file**: each line in this file represents a users' click history. You need to set `history_size` parameter in the config file, which is the max number of user's click history we use. We will automatically keep the last `history_size` number of user click history, if user's click history is more than `history_size`, and we will automatically pad with 0 if user's click history is less than `history_size`. the format is : <br> \n",
    "`[Userid] [newsid1,newsid2...]`<br>\n",
    "e.g., `train_U1 N1,N2` <br> \n",
    "\n",
    "- **document feature file**: It contains the word and entity features for news articles. News articles are represented by aligned title words and title entities. To take a quick example, a news title may be: <i>\"Trump to deliver State of the Union address next week\"</i>, then the title words value may be `CandidateNews:34,45,334,23,12,987,3456,111,456,432` and the title entitie value may be: `entity:45,0,0,0,0,0,0,0,0,0`. Only the first value of entity vector is non-zero due to the word \"Trump\". The title value and entity value is hashed from 1 to `n` (where `n` is the number of distinct words or entities). Each feature length should be fixed at k (`doc_size` parameter), if the number of words in document is more than k, you should truncate the document to k words, and if the number of words in document is less than k, you should pad 0 to the end. \n",
    "the format is like: <br> \n",
    "`[Newsid] [w1,w2,w3...wk] [e1,e2,e3...ek]`\n",
    "\n",
    "- **word embedding/entity embedding/ context embedding files**: These are `*.npy` files of pretrained embeddings. After loading, each file is a `[n+1,k]` two-dimensional matrix, n is the number of words(or entities) of their hash dictionary, k is dimension of the embedding, note that we keep embedding 0 for zero padding. \n",
    "\n",
    "In this experiment, we used GloVe\\[4\\] vectors to initialize the word embedding. We trained entity embedding using TransE\\[2\\] on knowledge graph and context embedding is the average of the entity's neighbors in the knowledge graph.<br>\n",
    "\n",
    "## MIND dataset\n",
    "\n",
    "MIND dataset\\[3\\] is a large-scale English news dataset. It was collected from anonymized behavior logs of Microsoft News website. MIND contains 1,000,000 users, 161,013 news articles and 15,777,377 impression logs. Every news article contains rich textual content including title, abstract, body, category and entities. Each impression log contains the click events, non-clicked events and historical news click behaviors of this user before this impression.\n",
    "\n",
    "A smaller version, [MIND-small](https://azure.microsoft.com/en-us/services/open-datasets/catalog/microsoft-news-dataset/), is a small version of the MIND dataset by randomly sampling 50,000 users and their behavior logs from the MIND dataset.\n",
    "\n",
    "The datasets contains these files for both training and validation data:\n",
    "\n",
    "#### behaviors.tsv\n",
    "\n",
    "The behaviors.tsv file contains the impression logs and users' news click hostories. It has 5 columns divided by the tab symbol:\n",
    "\n",
    "+ Impression ID. The ID of an impression.\n",
    "+ User ID. The anonymous ID of a user.\n",
    "+ Time. The impression time with format \"MM/DD/YYYY HH:MM:SS AM/PM\".\n",
    "+ History. The news click history (ID list of clicked news) of this user before this impression.\n",
    "+ Impressions. List of news displayed in this impression and user's click behaviors on them (1 for click and 0 for non-click).\n",
    "\n",
    "One simple example: \n",
    "\n",
    "`1    U82271    11/11/2019 3:28:58 PM    N3130 N11621 N12917 N4574 N12140 N9748    N13390-0 N7180-0 N20785-0 N6937-0 N15776-0 N25810-0 N20820-0 N6885-0 N27294-0 N18835-0 N16945-0 N7410-0 N23967-0 N22679-0 N20532-0 N26651-0 N22078-0 N4098-0 N16473-0 N13841-0 N15660-0 N25787-0 N2315-0 N1615-0 N9087-0 N23880-0 N3600-0 N24479-0 N22882-0 N26308-0 N13594-0 N2220-0 N28356-0 N17083-0 N21415-0 N18671-0 N9440-0 N17759-0 N10861-0 N21830-0 N8064-0 N5675-0 N15037-0 N26154-0 N15368-1 N481-0 N3256-0 N20663-0 N23940-0 N7654-0 N10729-0 N7090-0 N23596-0 N15901-0 N16348-0 N13645-0 N8124-0 N20094-0 N27774-0 N23011-0 N14832-0 N15971-0 N27729-0 N2167-0 N11186-0 N18390-0 N21328-0 N10992-0 N20122-0 N1958-0 N2004-0 N26156-0 N17632-0 N26146-0 N17322-0 N18403-0 N17397-0 N18215-0 N14475-0 N9781-0 N17958-0 N3370-0 N1127-0 N15525-0 N12657-0 N10537-0 N18224-0 `\n",
    "\n",
    "#### news.tsv\n",
    "\n",
    "The news.tsv file contains the detailed information of news articles involved in the behaviors.tsv file. It has 7 columns, which are divided by the tab symbol:\n",
    "\n",
    "+ News ID\n",
    "+ Category\n",
    "+ SubCategory\n",
    "+ Title\n",
    "+ Abstract\n",
    "+ URL\n",
    "+ Title Entities (entities contained in the title of this news)\n",
    "+ Abstract Entities (entites contained in the abstract of this news)\n",
    "\n",
    "One simple example: \n",
    "\n",
    "`N46466    lifestyle    lifestyleroyals    The Brands Queen Elizabeth, Prince Charles, and Prince Philip Swear By    Shop the notebooks, jackets, and more that the royals can't live without.    https://www.msn.com/en-us/lifestyle/lifestyleroyals/the-brands-queen-elizabeth,-prince-charles,-and-prince-philip-swear-by/ss-AAGH0ET?ocid=chopendata    [{\"Label\": \"Prince Philip, Duke of Edinburgh\", \"Type\": \"P\", \"WikidataId\": \"Q80976\", \"Confidence\": 1.0, \"OccurrenceOffsets\": [48], \"SurfaceForms\": [\"Prince Philip\"]}, {\"Label\": \"Charles, Prince of Wales\", \"Type\": \"P\", \"WikidataId\": \"Q43274\", \"Confidence\": 1.0, \"OccurrenceOffsets\": [28], \"SurfaceForms\": [\"Prince Charles\"]}, {\"Label\": \"Elizabeth II\", \"Type\": \"P\", \"WikidataId\": \"Q9682\", \"Confidence\": 0.97, \"OccurrenceOffsets\": [11], \"SurfaceForms\": [\"Queen Elizabeth\"]}]    [] `\n",
    "\n",
    "#### entity_embedding.vec & relation_embedding.vec\n",
    "\n",
    "The entity_embedding.vec and relation_embedding.vec files contain the 100-dimensional embeddings of the entities and relations learned from the subgraph (from WikiData knowledge graph) by TransE method. In both files, the first column is the ID of entity/relation, and the other columns are the embedding vector values.\n",
    "\n",
    "One simple example: \n",
    "\n",
    "`Q42306013  0.014516 -0.106958 0.024590 ... -0.080382`\n",
    "\n",
    "\n",
    "## DKN architecture\n",
    "\n",
    "The following figure shows the architecture of DKN.\n",
    "\n",
    "![](https://recodatasets.z20.web.core.windows.net/images/dkn_architecture.png)\n",
    "\n",
    "DKN takes one piece of candidate news and one piece of a user’s clicked news as input. For each piece of news, a specially designed KCNN is used to process its title and generate an embedding vector. KCNN is an extension of traditional CNN that allows flexibility in incorporating symbolic knowledge from a knowledge graph into sentence representation learning. \n",
    "\n",
    "With the KCNN, we obtain a set of embedding vectors for a user’s clicked history. To get final embedding of the user with\n",
    "respect to the current candidate news, we use an attention-based method to automatically match the candidate news to each piece\n",
    "of his clicked news, and aggregate the user’s historical interests with different weights. The candidate news embedding and the user embedding are concatenated and fed into a deep neural network (DNN) to calculate the predicted probability that the user will click the candidate news."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Global settings and imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "System version: 3.6.11 | packaged by conda-forge | (default, Aug  5 2020, 20:09:42) \n",
      "[GCC 7.5.0]\n",
      "Tensorflow version: 1.15.2\n"
     ]
    }
   ],
   "source": [
    "import sys\n",
    "sys.path.append(\"../../\")\n",
    "\n",
    "import os\n",
    "from tempfile import TemporaryDirectory\n",
    "import logging\n",
    "import papermill as pm\n",
    "import scrapbook as sb\n",
    "import tensorflow as tf\n",
    "tf.get_logger().setLevel('ERROR') # only show error messages\n",
    "\n",
    "from reco_utils.dataset.download_utils import maybe_download\n",
    "from reco_utils.dataset.mind import (download_mind, \n",
    "                                     extract_mind, \n",
    "                                     read_clickhistory, \n",
    "                                     get_train_input, \n",
    "                                     get_valid_input, \n",
    "                                     get_user_history,\n",
    "                                     get_words_and_entities,\n",
    "                                     generate_embeddings) \n",
    "from reco_utils.recommender.deeprec.deeprec_utils import prepare_hparams\n",
    "from reco_utils.recommender.deeprec.models.dkn import DKN\n",
    "from reco_utils.recommender.deeprec.io.dkn_iterator import DKNTextIterator\n",
    "\n",
    "print(f\"System version: {sys.version}\")\n",
    "print(f\"Tensorflow version: {tf.__version__}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Temp dir\n",
    "tmpdir = TemporaryDirectory()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "tags": [
     "parameters"
    ]
   },
   "outputs": [],
   "source": [
    "# Mind parameters\n",
    "MIND_SIZE = \"small\"\n",
    "\n",
    "# DKN parameters\n",
    "epochs = 10\n",
    "history_size = 50\n",
    "batch_size = 100\n",
    "\n",
    "# Paths\n",
    "data_path = os.path.join(tmpdir.name, \"mind-dkn\")\n",
    "train_file = os.path.join(data_path, \"train_mind.txt\")\n",
    "valid_file = os.path.join(data_path, \"valid_mind.txt\")\n",
    "user_history_file = os.path.join(data_path, \"user_history.txt\")\n",
    "infer_embedding_file = os.path.join(data_path, \"infer_embedding.txt\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data preparation\n",
    "\n",
    "In this example, let's go through a real case on how to apply DKN on a raw news dataset from the very beginning. We will download a copy of open-source MIND dataset, in its original raw format. Then we will process the raw data files into DKN's input data format, which is stated previously. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 51.7k/51.7k [00:08<00:00, 5.90kKB/s]\n",
      "100%|██████████| 30.2k/30.2k [00:05<00:00, 5.60kKB/s]\n"
     ]
    }
   ],
   "source": [
    "train_zip, valid_zip = download_mind(size=MIND_SIZE, dest_path=data_path)\n",
    "train_path, valid_path = extract_mind(train_zip, valid_zip)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "03:15:49 INFO: Train file /tmp/tmp3gwf53sv/mind-dkn/train_mind.txt successfully generated\n",
      "03:15:51 INFO: Validation file /tmp/tmp3gwf53sv/mind-dkn/valid_mind.txt successfully generated\n",
      "03:15:51 INFO: User history file /tmp/tmp3gwf53sv/mind-dkn/user_history.txt successfully generated\n"
     ]
    }
   ],
   "source": [
    "train_session, train_history = read_clickhistory(train_path, \"behaviors.tsv\")\n",
    "valid_session, valid_history = read_clickhistory(valid_path, \"behaviors.tsv\")\n",
    "get_train_input(train_session, train_file)\n",
    "get_valid_input(valid_session, valid_file)\n",
    "get_user_history(train_history, valid_history, user_history_file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_news = os.path.join(train_path, \"news.tsv\")\n",
    "valid_news = os.path.join(valid_path, \"news.tsv\")\n",
    "news_words, news_entities = get_words_and_entities(train_news, valid_news)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "03:15:53 INFO: Downloading glove...\n",
      "100%|██████████| 842k/842k [06:40<00:00, 2.10kKB/s] \n"
     ]
    }
   ],
   "source": [
    "train_entities = os.path.join(train_path, \"entity_embedding.vec\")\n",
    "valid_entities = os.path.join(valid_path, \"entity_embedding.vec\")\n",
    "news_feature_file, word_embeddings_file, entity_embeddings_file = generate_embeddings(\n",
    "    data_path,\n",
    "    news_words,\n",
    "    news_entities,\n",
    "    train_entities,\n",
    "    valid_entities,\n",
    "    max_sentence=10,\n",
    "    word_embedding_dim=100,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create hyper-parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "yaml_file = maybe_download(url=\"https://recodatasets.z20.web.core.windows.net/deeprec/deeprec/dkn/dkn_MINDsmall.yaml\", \n",
    "                           work_directory=data_path)\n",
    "hparams = prepare_hparams(yaml_file,\n",
    "                          news_feature_file=news_feature_file,\n",
    "                          user_history_file=user_history_file,\n",
    "                          wordEmb_file=word_embeddings_file,\n",
    "                          entityEmb_file=entity_embeddings_file,\n",
    "                          epochs=epochs,\n",
    "                          history_size=history_size,\n",
    "                          batch_size=batch_size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train the DKN model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "model = DKN(hparams, DKNTextIterator)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "step 10000 , total_loss: 0.4565, data_loss: 0.4562\n",
      "at epoch 1\n",
      "train info: logloss loss:0.4638135433827465\n",
      "eval info: auc:0.6705, group_auc:0.6381, mean_mrr:0.2957, ndcg@10:0.3883, ndcg@5:0.3237\n",
      "at epoch 1 , train time: 474.0 eval time: 620.1\n",
      "step 10000 , total_loss: 0.4472, data_loss: 0.4467\n",
      "at epoch 2\n",
      "train info: logloss loss:0.4505543474856317\n",
      "eval info: auc:0.6739, group_auc:0.639, mean_mrr:0.2946, ndcg@10:0.3879, ndcg@5:0.3218\n",
      "at epoch 2 , train time: 471.8 eval time: 612.2\n",
      "step 10000 , total_loss: 0.4424, data_loss: 0.4418\n",
      "at epoch 3\n",
      "train info: logloss loss:0.44420040403753597\n",
      "eval info: auc:0.6714, group_auc:0.6383, mean_mrr:0.294, ndcg@10:0.3872, ndcg@5:0.3204\n",
      "at epoch 3 , train time: 474.1 eval time: 617.7\n",
      "step 10000 , total_loss: 0.4466, data_loss: 0.4460\n",
      "at epoch 4\n",
      "train info: logloss loss:0.4390566331326689\n",
      "eval info: auc:0.6687, group_auc:0.639, mean_mrr:0.295, ndcg@10:0.3875, ndcg@5:0.3209\n",
      "at epoch 4 , train time: 474.5 eval time: 616.1\n",
      "step 10000 , total_loss: 0.4415, data_loss: 0.4408\n",
      "at epoch 5\n",
      "train info: logloss loss:0.4343439095525456\n",
      "eval info: auc:0.6672, group_auc:0.641, mean_mrr:0.2962, ndcg@10:0.388, ndcg@5:0.3217\n",
      "at epoch 5 , train time: 473.5 eval time: 608.2\n",
      "step 10000 , total_loss: 0.4351, data_loss: 0.4343\n",
      "at epoch 6\n",
      "train info: logloss loss:0.42958035677504997\n",
      "eval info: auc:0.6655, group_auc:0.639, mean_mrr:0.2946, ndcg@10:0.3858, ndcg@5:0.3199\n",
      "at epoch 6 , train time: 473.9 eval time: 610.2\n",
      "step 10000 , total_loss: 0.4348, data_loss: 0.4339\n",
      "at epoch 7\n",
      "train info: logloss loss:0.4246704682523697\n",
      "eval info: auc:0.6594, group_auc:0.6345, mean_mrr:0.2932, ndcg@10:0.3833, ndcg@5:0.3178\n",
      "at epoch 7 , train time: 472.9 eval time: 610.7\n",
      "step 10000 , total_loss: 0.4349, data_loss: 0.4338\n",
      "at epoch 8\n",
      "train info: logloss loss:0.41954958021519734\n",
      "eval info: auc:0.6551, group_auc:0.6321, mean_mrr:0.2921, ndcg@10:0.3815, ndcg@5:0.3165\n",
      "at epoch 8 , train time: 471.6 eval time: 606.7\n",
      "step 10000 , total_loss: 0.4386, data_loss: 0.4375\n",
      "at epoch 9\n",
      "train info: logloss loss:0.41416185889768203\n",
      "eval info: auc:0.6493, group_auc:0.6268, mean_mrr:0.2894, ndcg@10:0.3777, ndcg@5:0.3127\n",
      "at epoch 9 , train time: 473.2 eval time: 612.2\n",
      "step 10000 , total_loss: 0.4300, data_loss: 0.4288\n",
      "at epoch 10\n",
      "train info: logloss loss:0.40849729755098674\n",
      "eval info: auc:0.645, group_auc:0.6217, mean_mrr:0.2857, ndcg@10:0.3736, ndcg@5:0.3081\n",
      "at epoch 10 , train time: 473.5 eval time: 605.4\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<reco_utils.recommender.deeprec.models.dkn.DKN at 0x7fe60850deb8>"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(train_file, valid_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluate the DKN model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'auc': 0.645, 'group_auc': 0.6217, 'mean_mrr': 0.2857, 'ndcg@5': 0.3081, 'ndcg@10': 0.3736}\n"
     ]
    }
   ],
   "source": [
    "res = model.run_eval(valid_file)\n",
    "print(res)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/scrapbook.scrap.json+json": {
       "data": {
        "auc": 0.645,
        "group_auc": 0.6217,
        "mean_mrr": 0.2857,
        "ndcg@10": 0.3736,
        "ndcg@5": 0.3081
       },
       "encoder": "json",
       "name": "res",
       "version": 1
      }
     },
     "metadata": {
      "scrapbook": {
       "data": true,
       "display": false,
       "name": "res"
      }
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "sb.glue(\"res\", res)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Document embedding inference API\n",
    "\n",
    "After training, you can get document embedding through this document embedding inference API. The input file format is same with document feature file. The output file fomrat is: `[Newsid] [embedding]`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<reco_utils.recommender.deeprec.models.dkn.DKN at 0x7fe60850deb8>"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.run_get_embedding(news_feature_file, infer_embedding_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Results on large MIND dataset\n",
    "\n",
    "Here are performances using the large MIND dataset (1,000,000 users, 161,013 news articles and 15,777,377 impression logs). \n",
    "\n",
    "| Models | g-AUC | MRR |NDCG@5 | NDCG@10 |\n",
    "| :------| :------: | :------: | :------: | :------ |\n",
    "| LibFM | 0.5993 | 0.2823 | 0.3005 | 0.3574 |\n",
    "| Wide&Deep | 0.6216 | 0.2931 | 0.3138 | 0.3712 |\n",
    "| DKN | 0.6436 | 0.3128 | 0.3371 | 0.3908|\n",
    "\n",
    "\n",
    "Note that the results of DKN are using Microsoft recommender and the results of the first two models come from the MIND paper \\[3\\].\n",
    "We compare the results on the same test dataset. \n",
    "\n",
    "One epoch takes 6381.3s (5066.6s for training, 1314.7s for evaluating) for DKN on GPU. Hardware specification for running the large dataset: <br>\n",
    "GPU: Tesla P100-PCIE-16GB <br>\n",
    "CPU: 6 cores Intel(R) Xeon(R) CPU E5-2690 v4 @ 2.60GHz"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## References\n",
    "\n",
    "\\[1\\] Wang, Hongwei, et al. \"DKN: Deep Knowledge-Aware Network for News Recommendation.\" Proceedings of the 2018 World Wide Web Conference on World Wide Web. International World Wide Web Conferences Steering Committee, 2018.<br>\n",
    "\\[2\\] Knowledge Graph Embeddings including TransE, TransH, TransR and PTransE. https://github.com/thunlp/KB2E <br>\n",
    "\\[3\\] Wu, Fangzhao, et al. \"MIND: A Large-scale Dataset for News Recommendation\" Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. https://msnews.github.io/competition.html <br>\n",
    "\\[4\\] GloVe: Global Vectors for Word Representation. https://nlp.stanford.edu/projects/glove/"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Tags",
  "kernelspec": {
   "display_name": "Python (reco_gpu)",
   "language": "python",
   "name": "reco_gpu"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
